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PREFACE 



This manual describes performance utilities that run on the CRAY X-MP and 
CRAY-1 computer systems under the Cray operating system COS. They can be 
used with at least one of the following language processors: 

• Cray Fortran compiler CFT77 

• Cray Fortran compiler CFT 

• Cray Pascal compiler 

• Cray C compiler 

• Cray Assembly Language (CAL) assembler 

This manual describes the means of invoking these utilities, their 
options, and their output. 

Related publications from Cray Research, Inc. (CRI) are the following: 

SR-0018 CFT77 Reference Manual 

SR-0009 Fortran (CFT) Reference Manual 

SR-0222 CRAY X-MP Multitasking Programmer's Manual 

SR-2024 Cray C Reference Manual 

SR-0060 Pascal Reference Manual 

SR-2003 CAL Assembler Version 2 Reference Manual 

SR-0011 COS Version 1 Reference Manual 

SR-0112 Symbolic Debugging Package Reference Manual 

Within this manual, all publication references refer to CRI publications. 



CONVENTIONS 

This manual uses the following conventions: 

Convention Description 

italics Indicates variable information usually specified by 

the user or terms being defined 

[ ] Brackets enclose optional portions of a command or 

directive format 

bold Denotes a word from an output listing spelled as in 

the listing 



SR-0146 iii 



READER COMMENTS 

If you have any comments about the technical accuracy, content/ or 
organization of this manual/ please tell us. You have several options 
that you can use to notify us: 

• Call our Technical Publications department directly at 
(612) 681-5729 during normal business hours 

• Send us UNICOS or UNIX electronic mail at: 

ihnp4! cray! publications or sun! tundra! hall! publications 

• Use the Reader Comment form at the back of this manual 

• Write to us at the following address: 

Cray Research, Inc. 

Technical Publications Department 

1345 Northland Drive 

Mendota Heights, Minnesota 55120 

We value your comments and assure a prompt response. 
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INTRODUCTION 



This manual documents COS utilities for analyzing the performance of 
application programs. 

FTREF statically analyzes Fortran source code as an aid in understanding 
program structure, keeping track of variables and common blocks, and 
other aspects of programming. It is especially useful in the analysis of 
programs for multitasking. FTREF output includes a summary of common 
block references, a summary of entries and subprogram calls, and a static 
calling tree. 

FLOWTRACE monitors calls to and from all program units during execution 
of a program and prints a summary of these calls with statistics about 
execution time. By including in a program unit's timing the time used by 
its libary calls, it indicates the program unit's total execution time. 
FLOWTRACE output includes a summary of subprogram calls and timings and a 
calling tree. SETPLIMQ can be used to give separate timings for each 
call. 

SPY gives a statistical picture of the time spent in specific areas of 
code by monitoring the program address register during execution. The 
granularity of code areas to be monitored is selected by the user, and 
can be as small as a single word of instructions. This fine-grained 
monitoring differs from FLOWTRACE 's results at the level of program 
units. Although SPY'S results are approximate, they are useful for 
isolating inefficient code. SPY output includes a line for each 
specified code area, or bucket. Each line shows statistics on how many 
times one bucket was accessed, shown in the following ways: 

• An absolute count of accesses of the address bucket 

• The bucket's percentage of accesses within the routine 

• The bucket's percentage of accesses within the program 

Another table summarizes these results for entire routines rather than 
individual buckets. 

PERFMON monitors hardware performance during program execution and can be 
used for further analysis of execution speed. PERFMON output includes 
detailed information concerning scalar activity, hold issue conditions, 
memory use, or vectorization. 

This manual uses the terms program unit and subprogram in their 
Fortran senses, as defined in the reference manuals for the Cray Fortran 
compilers and by the American National Standards Institute (ANSI) 
standard X3. 9-1978, often called Fortran 77. For Fortran programs, both 
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terms refer to Fortran code that you compile, as distinguished from 
library routines or procedures written in other languages. A program 
unit can be a main program or a subprogram. A subprogram is either a 
subroutine, function, or block data subprogram. 

In other languages the term subprogram is equivalent to the following 
terms: 

• Pascal: procedure or function 

• C: function 

• CAL: entry 
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FTREF - ANALYZING FORTRAN 
SOURCE CODE 



The FTREF utility analyzes Fortran source listings and provides 
structural information about it. This analysis does not use information 
derived from executing a program. The analysis contains the following 
information: 

• Global cross-reference listing of common block variables 

• A static calling tree showing entry names, calling program units, 
and called subprograms for each subprogram 

• For multitasked applications, a summary of macrotasking 
subroutines used, with options to indicate whether a common 
variable or a subroutine is locked when it is referenced or 
redefined 

FTREF processes listings generated by either CFT77 or CFT. (CFT output 
should not be mixed with CFT77 output, however.) The dataset to be 
processed by FTREF may contain any number of program units used by the 
application. The more program units that are included in the dataset, 
the more complete will be the information output by FTREF. A listing can 
also contain output from multiple compilations using different levels of 
a compiler. 

FTREF is invoked by the FTREF control statement. A Fortran job must have 
been compiled previously, with the Fortran control statement specifying 
ON=XS to generate a cross-reference table and a source listing. The 
Fortran control statement must have specified a listing dataset with L=, 
and the FTREF control statement specifies an input dataset with I=. 
These should match unless you need to use different datasets, such as to 
combine more than one Fortran compilation into one dataset. 

The following examples show how to invoke FTREF in the same job as the 
compile step. These steps can be done in different jobs. 

JCL examples: 

JOB, . . . 

ACCOUNT, . . . 

CFT77 , ON=SX, L=LISTX. 

FTREF, I=LISTX,CB=FULL,TREE=FULL. 

/EOF 

CFT77 source 
/EOF 
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JOB, . . . 

ACCOUNT, . . . 

CFT,ON=SX,L=LISTX. 

FTREF / I=LISTX / CB=FULL / TREE=FULL. 

/EOF 

CFT source 
/EOF 

In the preceding examples, the compiler takes source code from the next 
file in the dataset and writes the source listing and cross-reference 
table in dataset LISTX; FTREF then uses LISTX as its input. The CB=FULL 
and TREE=FULL options represent a typical basic use of FTREF. 



2.1 FTREF CONTROL STATEMENT 

The FTREF control statement invokes FTREF to analyze listings from a 
previous Fortran compilation. The control statement can be in the same 
job as the compilation, as shown in the previous examples, or in a 
different job. The I parameter is required. 



Format: 



FTREF,I=idn,L=2dn,CB=op,TREE=op,R00T=r,END=end, 
LEVEL=n,DIR=dir,NORDER,MULTI. 



I=idn Input dataset containing listings generated by one or more 
Fortran compilations, in which the ON=SX options were 
specified. This parameter is required. Unless different 
datasets are needed, the I specification agrees with the L= 
dataset shown on the Fortran control statement. 

L=ldn Dataset to receive the output listing; the default is $OUT. 

CB=op Global common-block cross-references. The options are as 
follows: 

PART Identifies program units that use each common block; 
this is the default. 

FULL Details the use of common block variables in all 
program units. 

NONE No common block output 
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TREE=op Produces the static calling tree for the program and 
information about the subprograms called. The value 
**LOOP** in the output indicates there is an apparent 
recursive call in the program. The options are as follows: 

PART For each program unit, reports the entry names, 

external calls, names of program units that call it, 
and common block names; this is the default. 

FULL Reports the information that the PART option 

provides plus the static calling tree. The ROOT, 
END, and LEVEL parameters and the ROOT directive can 
be used to manage the calling tree output. See the 
examples at the end of this section. 

NONE No output information 

ROOT=r If TREE=FULL, ROOT causes subprogram r to be the root of 

the tree. Use ROOT when you want to see only one subtree of 
the calling tree. Use the ROOT directive (see subsection 
2.2.1) if you want to see more than one subtree. The 
default root is the main program; that is, the program unit 
that is not called. If there is more than one such program 
unit, the first by alphabetic order is used. 

END=end If TREE=FULL, END directs subprogram end to terminate any 
branch of the tree in which end is encountered. The value 
**STOP** is printed whenever the end subprogram is found, 
and that branch of the tree is terminated. By default, any 
calling tree generated by FTREF contains all subprograms in 
the program. 

LEVEL=n If TREE=FULL, LEVEL limits the length of any branch to n 
levels deep. The default is the entire program. If both 
LEVEL and END are specified, FTREF terminates a branch of 
the tree at whichever state is encountered first. 

DIR=dir Dataset dir contains a set of directives that control the 
processing (see subsection 2.2). If DIR is specified 
without a value, the directives are taken from $IN, whether 
the Fortran source is in $IN or another dataset. If DIR is 
not specified, no directives are read. 

NORDER Lists the source program's subprograms in input order 

instead of alphabetic order. The default is alphabetic 
order. 

MULTI Examines the source for uses of the macrotasking subroutines 
and generates tables summarizing the subroutines' use within 
the program. See the CRAY X-MP Multitasking Programmer's 
Manual, publication SR-0222. 
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2.2 DIRECTIVES 

FTREF directives are placed in a separate file; see the description of the 
DIR option on the FTREF control statement. 

A directive consists of a keyword and zero or more parameters. A blank, 
comma, or opening parenthesis separates the keyword and the parameters. 
The terminator is a period, closing parenthesis, or two consecutive 
blanks. 

A caret at the end of the directive line indicates that the next line is a 
continuation of the current directive. Do not precede the caret by a 
blank; it must immediately follow the last nonblank character of the line. 

Each line should contain only one directive. FTREF processes 80 
characters per line and ignores the rest of the characters. 



Examples : 

The following examples show the use of FTREF with a directive file. In 
both examples, FTREF is invoked in the same job with the compile step. 
The JCL, source file, and directive file are all in $IN. The DIR keyword 
on the FTREF control statement causes a directive file to be read from $IN 
(in these cases, the file following the source file). This file contains 
the CHKBLK directive. 



JOB, . . . 

ACCOUNT,.. . 

CFT77 , ON=SX, L=SOURCE . 

FTREF, I=SOURCE,CB=FULL,TREE=FULL, DIR. 

/EOF 

CFT77 source 
/EOF 
CHKBLK 
/EOF 



JOB, . . . 

ACCOUNT, . . . 

CFT, ON=XS , L=SOURCE . 

FTREF, I=SOURCE / CB=FULL,TREE=FULL, DIR. 

/EOF 

CFT source 
/EOF 
CHKBLK 
/EOF 
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2.2.1 ROOT DIRECTIVE 

The ROOT directive specifies a subprogram to be used as the root of a 
static calling tree (TREE=FULL). The output for each subprogram 
(specified in this directive or on the ROOT parameter) is a separate 
tree. Default processing is described with the ROOT parameter. 



ROOT, mdi , md 2 / . . . md n . 



md n Name of a root subprogram 



2.2.2 SUBSET DIRECTIVE 

The SUBSET directive specifies which program units will be processed by 
FTREF and prevents FTREF from processing program units not on the 
parameter list. If no SUBSET directive appears, or only one SUBSET 
directive is specified and it has no parameters, FTREF processes all 
program units in the input file. 



SUBSET, md 1 ,md2, . . .md n . 



mdf Name of the program unit to be processed by FTREF 



2.2.3 CHKBLK DIRECTIVE 

The CHKBLK directive specifies the common blocks whose variables FTREF 
will check to determine if the variables appear only in locked areas in a 
multitasked program. If no CHKBLK directive appears, FTREF does not check 
on whether variables are in a locked area. If there is only one CHKBLK 
directive and it is without a parameter, FTREF processes all the common 
block variables. 

A locked area begins with a LOCKON call and ends with a LOCKOFF call, both 
with the same lock variable. If a common block variable is referenced or 
redefined in such an area, it is considered to be locked; otherwise, it is 
unlocked. If a variable reference is unlocked, check to ensure that this 
is a safe condition. A locked area must be completely within a program 
unit to be detected by FTREF. Refer to the CRAY X-MP Multitasking 
Programmer's Manual, publication SR-0222, concerning the locking feature. 
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CHKBLK, 2)2*!, blk 2 /. . .2>2* n< 



blk 



Name of the common block whose variables FTREF will check to 
determine if a variable is in a locked area. To specify 
blank common, use // . 



NOTE 



FTREF searches for the presence of the LOCKON and 
LOCKOFF calls. It does not attempt a flow analysis; 
that is, it does not consider the effects of IF or GOTO 
statements on locks or variables. 



2.3 OUTPUT EXAMPLE 

In the following example of FTREF output, longer lines have been truncated 
to fit the page. The global common block reference is an excerpt of 
longer output. 



********************************************** 

* * 

* SUMMARY OF COMMON BLOCK REFERENCES * 

* * 
********************************************** 



BLOCK NAME 

#DA IS USED BY: 



// 



IS USED BY: 



SMATRIX IS USED BY: 



BKSUB DIFEQ PINVS 
DIFEQ SFROID 
DIFEQ PINVS RED 



PLGNDR RED SFROID SOLVDE 
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******************************************* 

* * 

* GLOBAL COMMON BLOCK CROSS REFERENCE * 

* * 
******************************************* 



BLOCK NAME 
#DA 

ADDRESS NAME TYPE MAIN USAGE MODULE 
246 C R 3DIM ARRAY SFROID 



SOURCE PROGRAM REFERENCE 
52P 4D 



1041 CONV R VARIABLE SFROID 

1044 I I VARIABLE SFROID 

50 INDEXV I 1DIM ARRAY SFROID 

50 NVARS I VARIABLE SOLVDE 



52P 


9S 


28U/2 


271 


52P 


22S 


58U 


8S 



21S 



20S 



// 

ADDRESS 


NAME 


TYPE MAIN USAGE 


MODULE 


55 


ANORM 


R 


VARIABLE 


SFROID 


55 


ANORM 


R 


VARIABLE 


DIFEQ 


54 


C2 


R 


VARIABLE 


SFROID 


54 


C2 


R 


VARIABLE 


DIFEQ 


51 


H 


R 


VARIABLE 


SFROID 


51 


H 


R 


VARIABLE 


DIFEQ 


52 


MM 


I 


VARIABLE 


SFROID 


52 


MM 


I 


VARIABLE 


DIFEQ 


SMATRIX 










ADDRESS 


NAME 


TYPE MAIN USAGE 


MODULE 





S 


R 


2DIM ARRAY 


PINVS 





S 


R 


2 DIM ARRAY 


RED 





S 


R 


2DIM ARRAY 


DIFEQ 



SOURCE PROGRAM REFERENCE 
46P 45P 42U 
26U 3D 



53U 


51U 


50S 


35U 


22U 


19U 


33U 


US 


3D 


48U 


34U 


32U 


53U/3 


44U 


43U/2 


27N 


25U 


15U 


49U 


37U 


22U 



SOURCE PROGRAM REFERENCE 
64U 54S 52U/2 52S 
15U/2 15S 10U/2 10S 
50S 49S 48S 47S 



ABBREVIATION USED ABOVE (KEY TO THE LINE NUMBER) 

D DEFINED IN DECLARATIVE STATEMENT 

N NAME USED AS A DO LOOP PARAMETER 

P ARGUMENT IN CALL OR FUNCTION CALL 

S STORED SO CONTENTS MAY BE CHANGED 

U NAME USED IN EXECUTABLE STATEMENT 

* DEFINED OR DECLARED BUT NEVER USED 

/2 REFERENCED MORE THAN ONE TIME IN THE STATEMENT 
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BKSUB 



************************************************ 

* * 

* SUMMARY OF ENTRIES AND ROUTINE CALLS * 

* * 
************************************************ 





ENTRY 


: BKSUB 






CALLS 


: ** NONE 


** 




CALLED BY 


: SOLVDE 






BLOCK USED 


: #DA 




DIFEQ 










ENTRY 


: DIFEQ 






CALLS 


** NONE 


** 




CALLED BY 


: SOLVDE 






BLOCK USED 


#DA 


// SMATRIX 


PINVS 










ENTRY 


PINVS 






CALLS 


** NONE 


** 




CALLED BY 


SOLVDE 






BLOCK USED 


#DA 


SMATRIX 


PLGNDR 










ENTRY 


. PLGNDR 






CALLS 


** NONE 


** 




CALLED BY 


: SFROID 






BLOCK USED 


#DA 




RED 










ENTRY 


RED 






CALLS 


** NONE 


** 




CALLED BY 


SOLVDE 






BLOCK USED. 


#DA 


SMATRIX 


SFROID 










ENTRY 


SFROID 






CALLS 


PLGNDR 


SOLVDE 




CALLED BY 


** NONE 


** 




BLOCK USED 


#DA 


// 


SOLVDE 










ENTRY 


SOLVDE 






CALLS 


BKSUB 


DIFEQ PINVS 




CALLED BY 


SFROID 






BLOCK USED 


#DA 





RED 
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******************************************* 

* * 

* STATIC CALLING TREE FOR SFROID * 

* * 
******************************************* 



SFROID 


PLGNDR 




< 1 > 


SOLVDE 


BKSUB 




< 2 > 


DIFEQ 




< 2 > 


PINVS 




< 2 > 


RED 
****** END OF THE CALLING TREE ****** 



Notes on the preceding output 

The preceding calling tree would look as follows in graphic form: 

SFROID 



PLGNDR 



SOLVDE 



BKSUB 



DIFEQ 



PINVS 



RED 



Another calling tree example: 



MAIN 


SUB1 


SUBSUB1 


SUB 2 2 


SUBSUB2 




< 2 > 


SUBSUB2 






< 1 > 


SUB 2 


SUBSUB1 


SUB 2 2 


SUBSUB2 


< 1 > 


SUB 3 









The above calling tree would look as follows in graphic form! 



MAIN 



SUB1 



SUBSUB1 

I 

SUB 2 2 

I 

SUBSUB2 



SUBSUB2 



SUB 2 
SUBSUB1 

I 

SUB 2 2 

I 

SUBSUB2 



SUB 3 
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FLOWTRACE - TRACING 
SUBPROGRAM CALLS 



FLOWTRACE generates printed information about all subprogram calls in a 
program; the output is based on execution of the program. The summary is 
written to dataset $OUT. The following information is shown: 

• The time spent in each program unit: amount, percentage of the 
total execution time, and average time per call 

• Number of calls to each subprogram and total calls in the program 

• Lists of subprograms that call and are called by each subprogram 

• A calling tree of the main program and all subprograms 



3.1 ENABLING AND DISABLING FLOWTRACE 

Flowtracing is activated by compiler options or directives, as follows: 

CFT77 or CFT: 

The main program must begin with a PROGRAM statement. 
Flowtracing can be enabled for an entire program or for parts of 
a program, as follows: 

- To trace an entire program, include ON=F on the CFT77 or CFT 
control statement. 

To trace specific parts of a program, insert directives in 
the source code: CDIR$ FLOW enables, and CDIR$ NOFLOW 
disables. In CFT77, these directives must be within a 
program unit. In CFT, FLOW must precede a program unit's 
header statement, and NOFLOW must follow its END statement. 

C: The -F option on the CC control statement applies to the whole 

executable program. See the following text concerning libraries. 

Pascal: The BT+ option can appear on the PASCAL control statement to 
apply to the whole program, or within compiler directives to 
enable flowtracing for specific parts of a program. Used within 
compiler directives, BT+ is disabled by BT-. See the following 
text concerning libraries. 
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When you run FLOWTRACE with C or Pascal, you must use the STACK or MULTI 
versions of libraries to be accessed. The default version of the 
libraries is site -dependent, as is the manner of selecting another 
version. Consult your site analyst about the default and the method of 
selection. 

It may be useful to disable flowtracing for small, frequently called 
subprograms because the FLOWTRACE overhead time can exceed the execution 
time. 

When flowtracing is enabled, a FLOWTRACE summary is listed after 
execution. Programs that terminate with library calls such as EXIT or 
ABORT must be modified for flowtracing: these calls should be replaced by 
END or STOP statements. 

Time spent in a lower-level called subprogram for which flowtracing is 
enabled is not counted as time spent in the calling subprogram. Time 
spent in library routines (such as SIN, PRINT, or CFFT) or in any 
subprogram for which flowtracing is not enabled is counted as time spent 
in the calling subprogram; these called subprograms are not listed in the 
summary. 

The output headed "Alphabetized summary" shows the following information: 

Column Description 



(leftmost) 



Routine 



Index into the calling tree for the first 999 
subprograms; *** indicates subsequent subprograms. 

Subprogram name, such as the name in a FUNCTION or 
SUBROUTINE statement 



Time executing 



Seconds of CPU time and percentage of the program's 
total time. The > symbol indicates a time less than 
1 ms; >>> means less than 1 microsecond. 



Called 
Average T 



Number of calls to the subprogram 

Average time per call to the subprogram. The > 
symbol indicates a time less than 1 ms; >>> means 
less than 1 microsecond. 



(next right) 
(rightmost) 



Absolute address of the subprogram 

Calling subprograms. If nothing follows the phrase 
"Called by," the subprogram was not called; normally 
this indicates the main program or the system. See 
the line for SFROID in the following example. 
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FLOWTRACE displays only the first 28 routines that called a given 
routine. If you require more than 28, ask your site analyst to modify the 
source for FLOWTRACE in $UTLIB, and rebuild it for you. 



Example (longer lines truncated): 



FLOWTRACE — Alphabetized summary 



Routine 


Time 


executing 


Called 


Aver; 


age T 


7 BKSUB 


0.007 


( 12.49%) 


15 




> 


4 DIFEQ 


0.002 


( 3.47%) 


630 




> 


5 PINVS 


0.029 


( 53.21%) 


630 




> 


2 PLGNDR 


> 


( 0.67%) 


120 




> 


6 RED 


0.007 


( 13.30%) 


615 




> 


1 SFROID 


0.002 


( 3.50%) 


1 




0.002 


3 SOLVDE 


0.007 


( 13.36%) 


4 




0.002 


TOTAL 


0.054 




2015 


Total 


calls 



@00003361a Called by SOLVDE 
<?00004717a Called by SOLVDE 
@00004044a Called by SOLVDE 
©00002340a Called by SFROID 
(300004431a Called by SOLVDE 
0.002 <?00002012a Called by 
0.002 @00003006a Called by SFROID 



The output headed "Calling tree" displays the dynamic calling tree by 
progressively indenting each successive level of the tree. 



Example: 



FLOWTRACE — Calling tree 



SFROID 00002012a 
PLGNDR 00002340a 



SOLVDE 
DIFEQ 
PINVS 
RED 
BKSUB 



00003006a 
00004717a 
00004044a 
00004431a 
00003361a 



The above tree is shown graphically at the end of section 2 



3.2 FLODUMP UTILITY: FOR ABORT CONDITIONS 

The FLODUMP utility recovers and dumps FLOWTRACE tables when a program 
aborts. It is activated by the FLODUMP control statement in the program's 
JCL, and should be included only if flowtracing is enabled (see subsection 
3.1). FLODUMP also requires a previous DUMPJOB statement. The following 
example illustrates the use of the FLODUMP control statement: 
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JOB , • • • . 

CFT77,ON=SF. 

SEGLDR,GO. 

EXIT. 

DUMP JOB . 

FLODUMP. 



Format: 



FLODUMP [,L=Idn]. 



L=ldn Dataset to write the report to; the default is $OUT. 



3.3 SETPLIMQ SUBROUTINE 

Subroutine SETPLIMQ enables FLOWTRACE to print a line on the output 
listing file for every CALL or RETURN statement executed, listing timing 
information per call rather than the total per subprogram. Because this 
option can generate a large volume of output, source code directives FLOW 
and NOFLOW should be used to include only areas of special interest. 

SETPLIMQ is requested by a subroutine call in the source code. In Fortran 
this call would appear as follows. 

CALL SETPLIMQ( COUnt) 

In general, the above statement should be one of the first executable 
statements in a program. The value of count specifies the number of 
trace lines printed. Since one line is produced for each CALL and each 
RETURN, count should be set to twice the number of CALL statements for 
which flowtracing is desired. 

In effect, each CALL and each RETURN statement is given a sequence number 
at run time; if this number is less than ABS(count) for a given CALL or 
RETURN, a line is printed to $OUT; these lines are interspersed with other 
output. CALL or RETURN statements executed before CALL SETPLIMQ are 
counted toward the line limit but generate no output. 
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SETPLIMQ output example: 



FLOWTRACE - 


- Call 


SOLVDE 


by 


SFROID 


407727978 


407301664 


FLOWTRACE - 


- Call 


DIFEQ 


by 


SOLVDE 


407744474 


407743808 


FLOWTRACE - 


Return 


DIFEQ 


to 


SOLVDE 


149 


149 


FLOWTRACE - 


- Call 


PINVS 


by 


SOLVDE 


407775007 


407743808 


FLOWTRACE - 


Return 


PINVS 


to 


SOLVDE 


844 


844 


FLOWTRACE - 


- Call 


DIFEQ 


by 


SOLVDE 


407806315 


407743808 


FLOWTRACE - 


Return 


DIFEQ 


to 


SOLVDE 


382 


531 



The longer numbers such as 407727978 are reference times shown in CPU 
clock periods. Shorter numbers such as 149 indicate how many clock 
periods were used by each call. The shorter numbers in the left 
column indicate how many clock periods were actually charged to a 
subprogram; these numbers can be used to assess program performance. 
Shorter numbers in the right column include FLOWTRACE overhead. 
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SPY - TIMING SELECTED 
CODE SEGMENTS 



SPY indicates how much time is being spent in different segments of code 
within routines. Routines measured by SPY can be written in any language 
available on Cray computers running COS. An advantage of SPY is that it 
entails almost no overhead. (A routine here is any identifiable, 
executable code with distinct entry and end points, whether it is part of 
your program or contained in a library. ) 

When so reguested by SPY, COS reads the address of the instruction 
currently being executed. Addresses are grouped in buckets, whose size 
is selectable; each access of a bucket is called a hit. After 
execution, SPY outputs statistics for all the buckets, including a bar 
graph on which "spikes" indicate code where the most time is spent. 

Because of the time interval between samples, SPY'S results involve 
probability: in the worst case, an address could be executed many times 
but never be sampled by SPY. The TS and BS parameters on the SPY control 
statement change the address and time intervals to provide a check on the 
probability. 

SPY's output includes a line for each bucket that was hit more than a 
specified number of times, followed by totals organized by subroutine. 
Because the volume of this output can be considerable, the SUB keyword is 
provided to let you specify which routines are to be represented. 

SPY results contain an inherent error which is insignificant for most 
applications. However, if your application program is very short, the 
following sources of noise may be greater than the real information 
gathered: 

• Loading CSP to crack your program's control statement 

• Loading your program 

• Loading CSP to crack the control statement which specifies the 
SPY, POST action 

• Loading SPY, turning off the COS SPY activity and obtaining the 
raw data 

The execution time for most application programs is long enough to make 
these noise sources insignificant. 
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The following examples show how SPY is used in three typical situations 
In all three cases, the program being spied is named SUBJECT. 



CFT77 with SEGLDR 

CFT77,DEBUG. 

SEGLDR, CMD='ABS=SUBJECT' . 

SPY, PREP. 

SUBJECT. 

SPY, POST. 



* Loader does not run program 

* Program executes 



CFT with LDR 

CFT,ON=IZ. 
LDR,NX,AB=SUBJECT. 
SPY, PREP. 
SUBJECT. 
SPY, POST. 



* Loader does not run program 

* Program executes 



CAL with SEGLDR 

CAL,SYM. 

SEGLDR , CMD= ' ABS=SUB JECT ' 

SPY, PREP. 

SUBJECT. 

SPY, POST. 



4.1 REQUIREMENTS FOR RUNNING SPY 

In addition to normal compiling, loading, and executing of the subject 
program, the JCL needed for using SPY must accomplish the following: 

• Generate a debug symbol table for the subject program 

• Load and execute the subject program in two separate steps 

• Execute SPY twice: both before and after execution of the subject 
program 
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4.1.1 GENERATING THE SYMBOL TABLE 

SPY requires a symbol table for the subject program, generated by the 
appropriate language processor. The table is sent to dataset $BLD by 
default, from which it is copied by the loader into $DEBUG. The table is 
generated by different processors using the following control statement 
specifications : 

Processor Keyword 

CFT77 DEBUG 

CFT ON=IZ or DEBUG (see below) 

CC -Z (deferred implementation) 

PASCAL 0=DM3 

CAL SYM 

Note that the DEBUG option on CFT turns off all optimization. This makes 
the result inapplicable for many situations. 



4.1.2 LOADING AND RUNNING THE SUBJECT PROGRAM 

When SPY is used, the subject program is loaded and run in separate 
steps. Between the load and run steps, SPY, PREP is run (see subsection 
4.1.3). The Cray loaders SEGLDR and LDR perform these separate steps as 
follows for program SUBJECT: 

SEGLDR, CMD='ABS=SUBJECT' . LDR,NX,AB=SUBJECT. 

• • • • • • 

SUBJECT. SUBJECT. 



4.1.3 THE PREP AND POST RUNS 

To use SPY you must include the SPY command both before and after the 
command line for the subject program, with the keywords PREP and POST, 
respectively. 

The SPY, PREP call reads the symbol table and calculates the absolute 
address of each symbol. Then, as specified by the TS, BS, and THRSHLD 
parameters, SPY performs one or more calls to COS, specifying regions of 
interest. Each call sets up a table so that at each interupt, COS reads 
the P register in your exchange package and increments the bucket that 
includes that P address. This information is written to dataset $SPY. A 
SPY, POST call then reads $SPY and generates a report as specified by the 
SUB, THRSHLD, and NOLIB parameters. 
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4.2 SPY CONTROL STATEMENT 



The format for the PREP call of SPY is as follows: 



SPY, PREP [,BS-bcktsz] [,D=dbugdn] [,S=scrtch] 
[ / SUB=rtn 1 :rtn 2 : . . .rtn n ] [,THRSHLD =gap] [,TS=time] 



BS=bcktsz Bucket size in words; default, 4. Each bucket begins on a 
word address that is a multiple of the bucket size. This 
parameter affects storage requirements; see subsection 4.3. 

D=dbugdn Dataset containing the symbol table for the subject program. 
The default is $DEBUG, which is the dataset to which SEGLDR 
and LDR write the symbol table (after copying it from $BLD). 

S=scrtch Dataset where the tables prepared by SPY, PREP are to be 

written, and from which SPY, POST is to read these tables. 
Default is $SPY. 

SUB=rtn 1 : rtn 2 : . . . rtn n 

List of routines that are to be reported on; n<20 . The 
default is to report on the entire executable program; that 
is, the SUB= parameter is used to limit output. This 
parameter affects storage requirements; see subsection 4.3. 

THRSHLD=grap 

This keyword is not needed when SEGLDR is used. It is used 
to make the best use of storage when LDR is used; see 
subsection 4.3. 

TS=time Time slice in microseconds; default, 500. 



The format for the POST call of SPY is as follows: 



SPY, POST [, ADDRESS] [,L=listdn] [,NOLABEL] [,NOLIB] 
[,S=scrtch] [,SUB=rtn 1 :rtn 2 :...rtn n ] [,MlNHlT=n]. 



ADDRESS Causes a report by address rather than by label 

L=listdn Dataset on which the final report is written; default, $OUT, 
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NOLABEL Causes the bucket size to be an entire routine. This makes 
the output similar to that of FLOWTRACE, except that time 
spent in library calls is accounted for separately. 

NOLIB Excludes library calls by eliminating any reference to a 
routine name that begins with a dollar sign ($). Library 
calls that do not begin with $, such as SECOND, are not 
affected by the NOLIB keyword. 

S=scrtch Dataset where the tables prepared by SPY, PREP are to be 

written, and from which SPY, POST is to read these tables. 
Default is $SPY. 

MINHIT=n Minimum number of hits required for an output line to be 

printed for a bucket or label; default 1. (MINHIT=0 is not 
recommended. ) 

S\]B=rtn 1 : rtn 2 : . . . rtn n 

List of routines that are to be reported on; n<_20. The 
default is to report on the entire executable program; the 
SUB= parameter is used to limit output. This parameter 
affects storage requirements; see subsection 4.3. 



4.3 STORAGE REQUIREMENTS 

r 
Using SPY causes COS to allocate storage in the Job Table Area (JTA) of 
your job. This space counts against the maximum memory allowed to your 
job. Smaller bucket sizes require more JTA memory than larger sizes. 

Several factors influence the amount of JTA storage required by a given 
symbol table: 

BS One JTA word is required for each bucket monitored in the 

user field. Larger bucket sizes provide less resolution but 
require less JTA space. The default is four words. 



THRSHLD=£ 



When you use LDR, areas of memory for code are intermingled 
with those for data. Since spying in data areas serves no 
purpose, the system jumps past any data area as large as t 
words (default 512), to the next code area. This saves the 
JTA space that would be used for buckets in the data area, 
but requires more JTA space for a separate call to SPY. 
THRSHLD allows you to adjust t for individual cases, in 
order to use the least JTA space. 

If this is a significant factor in your program, you should 
consider switching to SEGLDR, which gathers all code areas 
together. 
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SUB The SUB parameter can be used to limit SPY activity to 20 or 
fewer subprograms. Provided that using such a subset gives 
you the information you need, you can greatly reduce the JTA 
requirements this way. 

The default choices for the preceding parameters almost always yield a 
successful run, so they seldom need adjustment. However, if you encounter 
memory limitations, adjusting these parameters should allow you to get the 
information you need. 



4.4 SPY OUTPUT 

The main SPY output shows seven columns, giving the following information: 

Sample SPY output: 



ROUTINE 


LABEL 


ADDRESS 


HITS 


%PRG 


%SUB 


SECONDS 


CUM% 




MAIN 


MAIN 


00000201 


12 


7.6 


100.0 


0.006 


7.6 


******* 


SUB1 


110A 


00001400 


13 


8.3 


44.8 


0.006 


15.9 


******** 


SUB1 


306 


00001512 


6 


3.8 


20.7 


0.003 


19.7 


*** 


SUB1 


00003 


00001541 


3 


1.9 


10.3 


0.001 


21.7 


* 


SUB1 


115 


00001604 


3 


1.9 


10.3 


0.001 


23.6 


* 


SUB1 


HOD 


00001615 


1 


0.6 


3.4 


0.000 


24.2 




SUB1 


HOB 


00001620 


3 


1.9 


10.3 


0.001 


26.1 


* 


SUBZZZ 


SUBZZZ 


00003177 


4 


2.5 


80.0 


0.002 


28.7 


** 


SUBZZZ 


40 


00003240 


1 


0.6 


20.0 


0.000 


29.3 




$CDCO 


$CDCO 


00004140 


4 


2.5 


100.0 


0.002 


31.8 


** 


$IBMO 


$IBMO 


00004561 


1 


0.6 


100.0 


0.000 


32.5 




$RLD 


$RLD 


00007104 


1 


0.6 


100.0 


0.000 


33.1 




$WLD 


$WLD 


00013162 


2 


1.3 


100.0 


0.001 


34.4 


* 


$NICV 


$NICV 


00021640 


6 


3.8 


100.0 


0.003 


38.2 


*** 


$NOCV 


$NOCV 


00022217 


5 


3.2 


100.0 


0.002 


41.4 


*** 


$STOP 


$STOP 


00022700 


4 


2.5 


100.0 


0.002 


43.9 


** 


$GTDSP 


$GTDSP 


00024321 


10 


6.4 


100.0 


0.005 


50.3 


****** 


$RCW 


$RCW 


00025100 


2 


1.3 


100.0 


0.001 


51.6 


* 


$WCH 


$WCH 


00034452 


3 


1.9 


100.0 


0.001 


53.5 


* 
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Header Meaning 

ROUTINE Name of the routine using this address. Note that routine 
names beginning with $ are library routines and are 
documented in the Programmer ' s Library Reference Manual and 
the System Library Reference Manual, CRI publications 
SR-0113 and SM-0114. Many of these names suggest their 
functions, mathematical or otherwise. Routines whose names 
begin with $W and $R are often for writing and reading, 
respectively. 

LABEL The label used by SPY to identify the bucket. The label 
can be any of the following: 

The name of the program unit. This label is given to 
the first bucket in each subprogram. 

An integer followed by an alphabetic character. The 
integer is the label for a DO loop, and the character 
is used to identify more than one bucket in a single 
DO loop; for example, 11B is the label for the second 
bucket for the DO 11 loop in a program unit. 

A five-digit integer. SPY generates these labels for 
buckets to which neither of the above cases applies. 

ADDRESS Octal address of the bucket's first word 

HITS Number of hits for this bucket 

<*>PGM Percent of hits in program: hits for this label divided by 
total hits in the program, multiplied by 100 

%SUB Percent of hits in subroutine: hits for this label divided 
by total hits in subroutine 

SECONDS Estimated time: number of hits multiplied by time slice 

CUM% Cumulative sum of %PRG column. 

Far right Bar graph showing column 5, %PRG, percentage of hits in the 
program 
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Sample: 

ROUTINE SUMMARY 



HITS %PRG 



SECONDS 



MAIN 

SUB1 

SUBZZZ 

$CDCO 

$IBMO 

$RLD 

$WLD 

$NICV 

$NOCV 

$STOP 

$GTDSP 

$RCW 

$ATAN2 

$CABS 

$DASS 

UNDER 

BETWEEN 

OVER 

RUN SUMMARY 



12 


7.6 


29 


18.5 


5 


3.2 


4 


2.5 


1 


0.6 


1 


0.6 


2 


1.3 


6 


3.8 


5 


3.2 


4 


2.5 


10 


6.4 


2 


1.3 


7 


4.5 


27 


17.2 


1 


0.6 





0.0 





0.0 


20 


12.7 


157 





0.006 
0.014 
0.002 
0.002 
0.000 
0.000 
0.001 
0.003 
0.002 
0.002 
0.005 
0.001 
0.003 
0.013 
0.000 

0.000 
0.000 
0.010 
0.078 



%%%%%%%%%%%%%%%%%% 



% 



% 



%%%%%%%%%%%% 



After the information for all the buckets, a summary by routine is 
printed. The following columns are used: 



HITS 



Total hits for the routine 



%PRG Percent of hits in program: hits for this routine divided 
by total hits in program, multiplied by 100 

SECONDS Estimated time in seconds: number of hits multiplied by 
time slice 

Far right Bar graph showing column 3, %PGM, percentage of hits in the 
program 

At the bottom of the routine summary are the categories UNDER, BETWEEN, 
OVER, and RUN SUMMARY. The first three refer, respectively, to addresses 
that are lower than any area that was supposed to be measured, between 
such areas, or higher than these areas. If the UNDER and BETWEEN counts 
are significantly more than zero when the entire program is being 
measured, it indicates improper use of SPY. A high number for OVER can 
have various causes that do not indicate a problem, such as the use of 
overlays. The run summary is a total of all routines, including these 
final special categories. 

The estimated time should be similar to that generated by FLOWTRACE, with 
the qualification that FLOWTRACE includes time spent by library calls in 
the timing of the calling routine. 
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PERFMON - CRAY X-MP 
PERFORMANCE MONITOR 



PERFMON is an interface to the hardware performance monitor available on 
CRAY X-MP series computers. It is invoked within JCL. 

To use PERFMON, first execute it to turn on one of four available 
performance monitors. Then execute the program whose performance is to be 
monitored. To obtain the PERFMON report, execute PERFMON with the RESET 
or REPORT option immediately after the subject program. 

PERFMON results can be affected by the times taken for loading your 
program and loading CSP to crack your program's control statement. These 
are a factor only for very short applications. 



5.1 PERFMON CONTROL STATEMENT 

The format of the PERFMON control statement is as follows: 



PERFMON [,L=list] , result. 



L=listdn Specifies the listing output dataset for PERFMON reports. 
The default is $OUT. 

result Specifies the primary effect of the PERFMON control 

statement; can be one of the following: 

ON=n Turns on performance monitoring and selects one 

of four hardware performance monitors. These are 
indicated in PERFMON output at the upper right in 
each block as "Monitor 0" and so on. The 
monitors record the following: 

Execution summary 

1 Hold issue conditions 

2 Memory activity 

3 Vector events and instruction summary 

OFF Turns off performance monitoring 
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REPORT Produces a report of the current performance 
monitor counters (to the L listing file). It 
does not clear the counters. 

RESET Produces a report of the current performance 
monitor counters (to the L listing file). In 
addition, the counters are cleared after printing 
and monitoring is turned off. 



JCL example for program SUBJECT: 

SEGLDR,CMD= , ABS=SUBJECT' . 

PERFMON,ON=0. 

SUBJECT. 

PERFMON, RESET. 

PERFM0N / 0N=1. 

SUBJECT. 

PERFMON, RESET. 

PERFMON, 0N= 2. 

SUBJECT. 

PERFMON, RESET. 

PERFMON, 0N= 3. 

SUBJECT. 

PERFMON, RESET. 



5.2 PERFMON OUTPUT 

Each run of PERFMON (that is, each monitor used) generates one block of 
information. 



Output sample of monitor 0, execution summary: 



Cycles: 6173669, CP seconds: 0.058649856, Task number: 1, Monitor 0. 

36.95 Million instr/sec (MIPS) 2166976 instructions issued 

2.85 avg. clock periods per instruction. 

43.87 % of CP holding issue. 2708154 CP holding issue 

407861.19 instr. buf. fetches/sec. 23921 fetches 

0.00 I/O memory references/sec. I/O references 

6528183.18 CPU memory references/sec. 382877 CPU references 

2972522.24 Floating adds/sec. 174338 F.P. adds 

4045108.01 Floating multiplies/sec. 237245 F.P. multiplies 

710419.48 Floating reciprocals/sec. 41666 F.P. reciprocals 

7.73 Million Floating Operations/sec (MFLOPS) 
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Output sample of monitor 1, hold issue conditions: 



Cycles: 6302128, CP seconds: 



0.059870216, Task number: 



1, Monitor 1 



0. 


.00 


% 


CP 


hold 


issue 


0. 


.01 


% 


CP 


hold 


issue 


6. 


.26 


% 


CP 


hold 


issue 


36. 


.13 


% 


CP 


hold 


issue 


0. 


.16 


% 


CP 


hold 


issue 


0. 


.24 


% 


CP 


hold 


issue 


0. 


.47 


% 


CP 


hold 


issue 


0, 


.74 


% 


CP 


hold 


issue 



semaphores 

shared registers 

A-regs & A-functionals 

S-regs & S-functionals 

V-regs 

V-functionals 

scalar memory 

block memory 



Output sample of monitor 2, memory activity: 



Cycles: 6387292, CP seconds: 0.060679274, Task number: 1, Monitor 2 

412694.46 instr. buffer fetches/sec. 
4413154.32 scalar memory refs/sec. 

40.94 % conflicts/scalar reference 
33751.23 Block memory refs/sec. 

19.63 % conflicts/block reference 
2050156.37 Block memory refs/sec. 

18.51 % conflicts/block reference 
1303344.53 vector memory references/sec. 



Output sample of monitor 3, vector events and instruction summary: 



Cycles: 5069820, CP seconds: 0.048163290, Task number: 1, Monitor 3. 

4410496.04 000-017 instructions/sec issued, 
12.03 % of all instructions 
32123345.39 scalar (020-137) instructions/sec issued, 
87.64 % of all instructions 
14554.65 vector integer&logical instr. /sec. issued, 

0.04 % of all instructions 
37123.71 vector floating-point instr. /sec. issued, 

0.10 % of all instructions 
66772.85 vector load-store instr. /sec. issued, 
0.18 % of all instructions 
412409.53 vector integer operations/sec. 28.34 elements/instruction 
917337.67 vector floating operations/sec. 24.71 elements/instruction 
1555811.49 vector memory references/sec. 23.30 elements/instruction 
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ADDRESS column in SPY output, 4-6 

ADDRESS parameter, SPY, 4-4 

ANSI 77 terminology, 1-1 

Average T column, FLOWTRACE output, 3-2 



BETWEEN category in SPY output, 4-8 
Blank common, 2-6 
$BLD dataset, and SPY, 4-3, 4-4 
Block data subprogram, 1-2 
BS parameter, SPY, 4-1, 4-3 to 4-5 
BT Pascal option, with FLOWTRACE, 3-1 
Bucket, SPY, 4-1 
size, 4-4 



C, 1-2 

and FLOWTRACE, 3-1 
CAL, 1-2 

with SPY, 4-3 
Called column, FLOWTRACE output, 3-2 
Calling tree 

from FLOWTRACE, 3-3 

from FTREF, 2-1, 2-5 
CB keyword, FTREF, 2-2 
CC control statement 

with FLOWTRACE, 3-1 

with SPY, 4-3 
CFT77 or CFT (also see Fortran) 

with FLOWTRACE, 3-1 

with FTREF, 2-1 

with SPY, 4-3 
CHKBLK directive (FTREF), 2-5 
Common block 

Block data subprogram, 1-2 

blank, 2-6 

variable, 2-1, 2-5 

cross-references, FTREF, 2-2 
Cross-reference listing, from FTREF, 2-1 
CSP overhead 

SPY, 4-1 

PERFMON, 5-1 



D parameter, SPY, 4-4 

DEBUG Fortran keyword, with SPY, 4-3 

Debug symbol table, with SPY, 4-2 

$DEBUG, SPY, 4-4 

DIR keyword, FTREF, 2-3 



END keyword, FTREF, 2-3 



END statement, with FLOWTRACE, 3-1 

Entry, in CAL, 1-2 

Execution summary, from PERFMON, 5-2 



F Fortran option, with FLOWTRACE, 3-1 
-F option for C, with FLOWTRACE, 3-1 
FLODUMP utility (FLOWTRACE), 3-3 
FLOW compiler directive, and FLOWTRACE, 

3-1, 3-4 
FLOWTRACE, 3-1 

and other utilities, 1-1 

compared with SPY, 4-8 

enabling and disabling, 3-1 

FLODUMP utility, 3-3 

output, 3-2 

SETPLIMQ subroutine, 3-4 
Fortran 

DEBUG keyword, with SPY, 4-3 

F option, with FLOWTRACE, 3-1 

L keyword (listing dataset) 
and FTREF, 2-1 
and SPY, 4-4 

source, and FTREF, 2-1 

terms, 1-1 
FTREF, 2-1 

and other utilities, 1-1 

control statement, 2-2 

CHKBLK directive, 2-5 

ROOT directive, 2-5 

SUBSET directive, 2-5 
FULL option, FTREF, 2-2, 2-3, 2-5 
Function, in C, 1-2 



Hit, SPY, 4-1 

HITS column in SPY output, 4-6, 4-8 

Hold issue conditions, from PERFMON, 5-3 



I CFT option, with SPY, 4-3 
I keyword, FTREF, 2-1, 2-2 
$IN, for FTREF directives, 2-3 
Instruction summary, from PERFMON, 5-3 



Job Table Area (JTA), SPY, 4-5 



L keyword 

FTREF, 2-2 
PERFMON, 5-1 
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L Fortran keyword (listing dataset) 

and FTREF, 2-1 

and SPY, 4-4 
LABEL column in SPY output, 4-6 
LDR, and SPY, 4-3 
LEVEL keyword, FTREF, 2-3 
Libraries, with FLOWTRACE, 3-2 
Loading overhead, SPY, 4-1 
Locked areas, checked by FTREF, 2-1, 2-5 
LOCKON, LOCKOFF calls, 2-5 
**LOOP** value, FTREF, 2-3 



Macrotasking subroutines, and FTREF, 2-1, 

2-3 
Memory activity, from PERFMON, 5-3 
MINHIT keyword, SPY, 4-5 
MULTI keyword, FTREF, 2-3 
MULTI libraries, with FLOWTRACE, 3-2 
Multitasking, and FTREF, 2-1, 2-3, 2-5 



NOFLOW compiler directive, and FLOWTRACE, 

3-1, 3-4 
NOLABEL parameter, SPY, 4-5 
NOLIB parameter, SPY, 4-3, 4-5 
NONE option, FTREF, 2-2, 2-3 
NORDER keyword, FTREF, 2-3 



Pascal option, with SPY, 4-3 

ON and OFF options, PERFMON, 5-1 

ON=F Fortran option, with FLOWTRACE, 3-1 

ON=IZ CFT keyword, with SPY, 4-3 

$OUT dataset 

with FLOWTRACE, 3-1, 3-4 

with PERFMON, 5-1 

with SPY, 4-4 
OVER category in SPY output, 4-8 



ROOT (FTREF) 

directive, 2-5 

keyword, 2-3 
ROUTINE column in SPY output, 4-6 
Routine column in FLOWTRACE output, 3-2 
RUN SUMMARY category in SPY output, 4-8 



S parameter, SPY, 4-4, 4-5 

SECONDS column in SPY output, 4-7, 4-8 

SEGLDR, and SPY, 4-3, 4-4 

SETPLIMQ subroutine (FLOWTRACE), 3-4 

$SPY dataset, 4-3, 4-4 

SPY, 4-1 

and other utilities, 1-1 
compared with FLOWTRACE, 4-8 
control statement, 4-4 
generating symbol table, 4-3 
handling subject program, 4-3 
output, 4-6 
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